In digital cameras, a color separation filter for capturing color images is conventionally provided on an image sensor such as CCD (Charge Coupled Device) or MOS (Metal Oxide Semiconductor). Taking the Bayer array for instance, filters of primary colors, R (red), G (green), and B (blue) are arranged in a checkerboard pattern. Hereinafter, image data captured through the color separation filter is called RAW data. The RAW data has a larger file size than any data of JPEG format, and it is time-consuming to develop and reproduce the RAW data. As a result of higher pixels and ongoing developments of faster data-read techniques in image sensors, small RAW images smaller than source images are more often used these days. The small RAW images are paid attention for their use in; a monitor mode for monitoring photographic subjects on a liquid crystal monitoring device to capture still images, and HD movies such as moving images normalized in size. A conventional processing method for small images is to sum and read signals of the same color at every other pixel in an image sensor (pixel mixing reading method). This method provides smaller RAW data having a reduced number of output effective pixels, enabling a shorter processing time and energy saving. The typical examples of the pixel mixing method are a nine-pixel mixing method and a four-pixel mixing method.
Describing the nine-pixel mixing, three pixels in the horizontal direction and three pixels in the vertical direction for signals of the same color at every other pixel, that is matrix data of nine pixels in total, are mixed and used as data of one pixel to reduce an image size (data volume) to one-ninth of its original size. Describing the four-pixel mixing, two pixels in the horizontal direction and two pixels in the vertical direction for signals of the same color at every other pixel, that is matrix data of four pixels in total, are mixed and used as data of one pixel in order to reduce an image size (data volume) to one-fourth of its original size.
The nine-pixel mixing in the primary color Bayer array is briefly described referring to FIGS. 29 to 36. The symbols with small letters, b, g1, g2, and r, denote color elements before mixing, and symbols, B, G1, G2, and R, denote color elements after mixing. The color elements b and B are blue, g1 and g2 are green, and r and R are red. The color elements g1 and G1 are green laterally next to the color elements b and B. The color elements g2 and G2 are green vertically next to the color elements b and B. Based on the periodicity of the RGB Bayer array, bg- or BG-repeated line data and gr- or GR-repeated line data along the horizontal direction are alternately outputted in the vertical direction. The Bayer array before mixing remains unchanged after the pixels are mixed.
FIG. 29 illustrates an example of mixing the color elements b (blue). The quadrangles drawn with bold lines each represents a unit block made up by the pixel data of nine color elements b to be mixed in the distribution space of RAW data of a source image. In a unit block of the 1st row and 1st column, data of nine pixel data b illustrated with circles, which are respectively in the 1st row and 1st column, the 1st row and 3rd column, the 1st row and 5th column, the 3rd row and 1st column, the 3rd row and 3rd column, the 3rd row and 5th column, the 5th row and 1st column, the 5th row and 3rd column, and the 5th row and 5th column, are mixed by addition. As illustrated in FIG. 30, pixel data B obtained by the mixing is allocated so that a pixel at the center of three rows and three columns in horizontal and vertical directions carries the mixed pixel data B. The positions of the pixel data B illustrated in FIG. 30 are the positions of pixels of the same color b, which is a difference to the four-pixel mixing described later as is known from the comparison between FIG. 36 and FIG. 41. The pixel allocation is applied to all of the unit blocks, and pixels in the pixel data of the mixed color elements B (blue) are allocated as illustrated in FIG. 30.
FIG. 31 illustrates an example of mixing the color elements g1 (green). The quadrangles drawn with bold lines each represents a unit block made up by the pixel data of nine color elements g1 (green) to be mixed. In unit blocks illustrated in FIG. 31, pixels at the center in the lateral direction are located in the middle of two unit blocks adjacent to each other illustrated in FIG. 29. Referring to a unit block of the 1st row and 1st column illustrated in FIG. 31, data of nine pixels illustrated with circles, which are respectively in the 1st row and 4th column, the 1st row and 6th column, the 1st row and 8th column, the 3rd row and 4th column, the 3rd row and 6th column, the 3rd row and 8th column, the 5th row and 4th column, the 5th row and 6th column, and the 5th row and 8th column, are mixed by addition. As illustrated in FIG. 32, pixel data obtained by the mixing is allocated so that a pixel at the center of three rows and three columns in horizontal and vertical directions carries the mixed pixel data. The pixel allocation is applied to all of the unit blocks, and pixels in the pixel data of the mixed color elements G1 (green) are allocated as illustrated in FIG. 32.
As is clear from the drawing of FIG. 33 where the illustrations of FIGS. 30 and 32 are combined, coordinates of the allocated pixels in the pixel data of the mixed color elements B and the color elements G1 are uniformly distributed in the horizontal direction with intervals of two pixels in the horizontal direction between the different color elements. In the illustrations of FIGS. 30 and 32, there are intervals of an odd number of pixels, five pixels, in the horizontal direction between the same color elements. Therefore, pixels of different colors can be conveniently allocated in the middle of two adjacent pixels spaced from each other by five pixels. The odd number is expressed by (2m+1), where m is an arbitrary natural number. The middle position expressed by {(2m+1)+1}/2=m+1 (natural number) resulting in a dividable number is a right position for the allocation of such pixels.
FIG. 34 illustrates an example of mixing the color elements g2 (green). The coordinates of the allocated pixels in the mixed data are G2 in FIG. 36. FIG. 35 illustrates an example of mixing the color elements r (red). The coordinates of the allocated pixels in the mixed data are R in FIG. 36. FIG. 36 illustrates all of the coordinates of the allocated pixels in the B, G1, G2, and R mixed data. As is clearly known from FIG. 36, the color elements B and G2, and the color elements G1 and R aligned in the vertical direction are spaced from each other at equal intervals. There are intervals of two pixels between the different color elements, and there are intervals of five pixels between the same color elements. Therefore, pixels of different colors can be conveniently allocated in the middle of two adjacent pixels spaced from each other by five pixels. In the nine-pixel mixing, coordinates of the allocated pixels in the mixed data are uniformly distributed in the horizontal and vertical directions.
The four-pixel mixing is described referring to FIGS. 37 to 41. FIG. 37 illustrates an example of mixing the color elements b. In a unit block of the 1st row and 1st column, data of four pixel data b illustrated with circles, which are respectively in the 1st row and 1st column, the 1st row and 3rd column, the 3rd row and 1st column, and the 3rd row and 3rd column, are mixed by addition. As illustrated in FIG. 41, pixel data B obtained by the mixing is allocated so that a pixel at the center of two rows and two columns in horizontal and vertical directions carries the mixed pixel data B. The positions of the pixel data B illustrated in FIG. 41 are the positions of pixels of the color r in FIG. 37 (in contrast to the nine-pixel mixing where these positions are the pixel positions of the same color). The pixel allocation is applied to all of the unit blocks, and pixels of the pixel data of the mixed color elements B are allocated as illustrated in FIG. 41.
FIG. 38 illustrates an example of mixing the color elements g1. In the given example, data of four pixel data b illustrated with circles, which are respectively in the 1st row and 2nd column, the 1st row and 4th column, the 3rd row and 2nd column, and the 3rd row and 4th column, are mixed by addition. As illustrated in FIG. 41, pixel data obtained by the mixing is allocated so that a pixel at the center of two rows and two columns in horizontal and vertical directions carries the mixed pixel data. The positions of the pixel data G1 illustrated in FIG. 41 are the positions of pixels of the same color g2 in FIG. 38 (though different suffixes in G1 and g2). The pixel allocation is applied to all of the unit blocks, and pixels of the pixel data of the mixed color elements G1 are allocated as illustrated in FIG. 41.
FIG. 39 illustrates an example of mixing the color elements g2. The coordinates of the allocated pixels in the mixed data are G2 illustrated in FIG. 41. FIG. 40 illustrates an example of mixing the color elements r. The coordinates of the allocated pixels in the mixed data are R illustrated in FIG. 41. In the illustration of FIG. 41, the pixel data R are allocated at the pixel positions of the color b in FIG. 40 (in contrast to the nine-pixel mixing where these positions are the pixel positions of the same color).
As is learnt from the drawing of FIG. 41 where the illustrations of these pixel positions are combined, coordinates of the allocated pixels in the mixed data of the color elements B and the color elements G1 are not uniformly distributed in the horizontal direction. Similarly, coordinates of the allocated pixels in the mixed data of the color elements G2 and the color elements R are not uniformly distributed in the horizontal direction. On the other hand, coordinates of the allocated pixels in the mixed data of the color elements B and the color elements G2 are not uniformly distributed in the vertical direction, and coordinates of the allocated pixels in the mixed data of the color elements G1 and the color elements R are not uniformly distributed in the vertical direction.
FIGS. 42A to 42C are summarized illustrations of FIGS. 37 through 41. An image sensor of the primary color Bayer array has filters of the same colors arranged at every other pixel. When pixels are mixed in the horizontal direction, therefore, the image sensor is driven by timings corresponding to the every other pixel. The image sensor is driven likewise when the pixels are mixed in the vertical direction. FIG. 42A illustrates the source image RAW data. FIG. 42B illustrates the distribution of pixel-mixed RAW data on the source image RAW data, which is an illustration corresponding to FIG. 41. FIGS. 41 and 42B both illustrate the distribution of pixel data, indicating which of data at different coordinate positions on the source image is carried by each one of pixel data serially inputted. FIG. 42C illustrates a pixel arrangement of the RAW data where pixels are equally spaced timing-wise after four pixels are mixed.
To read the pixel data by mixing four pixels in the image sensor illustrated in FIG. 42A, BG-repeated line data of two lines are transmitted to a pixel mixer in a next stage from a photo detector (photoelectric converter) of the image sensor. The pixel mixer includes a vertical transfer switch, a signal voltage retainer circuit including capacitors, and a horizontal transfer switch. The pixel mixer outputs the BG-repeated line data after four pixels are mixed from a signal output line (see the Patent Reference 4). Then, GR-repeated line data of two lines are transmitted to the pixel mixer from the photo detector of the image sensor, and the pixel mixer outputs the GR-repeated line data after four pixels are mixed through the signal output line. The BG-repeated line data and the GR-repeated line data after four pixels are mixed are outputted serially in turns.
A group of pixels continuous in the pixel-repeated line data are equally spaced timing-wise. It may as well be said that, in the mixed pixel data of the Bayer array, timings by which data is serially inputted are timings at equal intervals in the horizontal direction, and are also timings at equal intervals in the vertical direction. However, the distribution of pixel data, indicating which of pixel at different positions on the source image is carried by each one of these pixel data, is not uniform as illustrated in FIGS. 41 and 42B. The lack of uniformity in the distribution of pixel data may be a factor that causes the degradation of an image quality as described below.
As a result of the four-pixel mixing, the coordinate position of a pixel where the mixed pixel data is allocated corresponds to a position in the middle of two pixels before the pixels are mixed. Referring to the illustrations of FIGS. 41 and 42B, the coordinate positions of pixels where the mixed pixel data are respectively allocated are not equally spaced in the vertical and horizontal directions. Horizontally and vertically, the odd-numbered B and G are very close to each other, the odd-numbered G and the even-numbered B are very distant from each other, the odd-numbered G and R are very close to each other, and the odd-numbered R and the even-numbered G B are very distant from each other. A pixel count equivalent to an interval between the very closely spaced color elements is zero, while a pixel count equivalent to an interval between the very distantly spaced color elements is two. Thus, the color elements are not equally spaced from one another. In the Bayer array after the pixels are mixed, the non-uniform distribution of pixel data leads to the loss of continuity of aliasing components to be obtained by the pixel mixing from high-frequency information to low-frequency information. This adversely affects an image quality.
The differences between the nine-pixel mixing and the four-pixel mixing are described below. In the nine-pixel mixing, the pixel data obtained by mixing the blue color elements are allocated on the odd-numbered lines both horizontally and vertically, and the pixel data obtained by mixing the red color elements are allocated on the even-numbered lines both horizontally and vertically. Thus, the lines where these pixel data are distributed remain unchanged before and after the pixel mixing. And, the same goes for the lines where the pixel data obtained by mixing the green 1 and green 2 color elements, respectively, are allocated, with the lines remaining unchanged likewise before and after the pixel mixing. In the four-pixel mixing, however, the lines of the pixel data obtained by mixing the blue color elements change from the odd-numbered lines to the even-numbered lines both horizontally and vertically, and the lines of the pixel data obtained by mixing the red color elements change from the even-numbered lines to the odd-numbered lines both horizontally and vertically. And, the same goes for the lines of the pixel data obtained by mixing the green 1 and green 2 color elements respectively, with the lines changing likewise before and after the pixel mixing.
A conventional art for correcting the distribution of pixel data is directed at solving problems such as the occurrence of image distortion and/or moire when signals of the same color are added in every other line and then read (see the Patent Reference 1). This conventional art is, however, designed to correct the distribution to be uniform in the vertical direction alone. The conventional art disclosed in the Patent Reference 1 is briefly described referring to FIGS. 43A to 43C.
As illustrated in FIG. 43A, an image sensor m2 is driven by timing signals outputted from a driver m8 of the image sensor. Of pixel signals of the same color, the pixel signals adjacent to each other in the vertical direction are mixed as illustrated in FIG. 43B (a) and (b). The resulting signal is converted to a digital signal by an AD converter m4. The digital signal is processed by a camera signal processor m6, for example, subjected to a color separation process. As a result of the process, a luminance signal and a color difference signal are generated. In the luminance signal and the color difference signal thus generated, lines are not equally spaced; a (2n−1)th line and a 2 nth line are very close to each other but the 2 nth line and a (2n+1)th line are very distant from each other, where n is a natural number (n=1, 2, . . . ).
As illustrated in FIG. 43C, a correction processor (center-of-gravity displacement correction processor) m10 corrects the non-uniform distribution of pixel data in the color difference signal and the luminance signal outputted from the camera signal processor m6. Lines Y2n and Y (2n±even number) are even-numbered lines of the color difference signal and the luminance signal. The line Y2n, Y (2n±even number) is an upper one of two lines very close to each other due to the vertically non-uniform distribution of pixel data. The lines Y (2n±odd number) are odd-numbered lines of the luminance signal and the color difference signal. The line Y (2n±odd number) is a lower one of the two very close lines. Lines Y′2n and Y′ (2n±even number) are even-numbered lines of the color difference signal and the luminance signal after the non-uniform distribution of pixel data is corrected. Lines Y′ (2n±odd number) are odd-numbered lines of the color difference signal and the luminance signal after the non-uniform distribution is corrected.
The ratio of an interval between Y2n and Y2n+1 to an interval between Y2n+1 and Y2n+2 is 1:3. Therefore, the correction process is performed so that the signals have an equal interval after the vertically non-uniform distribution of pixel data is corrected, as expressed by the following equations.Y′2n=Y2n Y′2n+1=(⅔)×Y2n+1+(⅓)×Y2n+2
The signals of the even-numbered lines are outputted as-is, and the signals of the odd-numbered lines are interpolated so that a ratio of these signals to those of the next lines is 2:1.
As a result of such a process, the coordinates of the pixels where the mixed pixel data are allocated are uniformly distributed in the vertical direction. An image thereby obtained has no image distortion or moire because the non-uniform distribution of pixel data has been corrected.
In order to record moving images normalized in size, the RAW data may be reduced in size to one-fourth by mixing the pixels on the image sensor and then subjected to a resizing process on the Bayer array (see an example disclosed in the Patent Reference 2). According to the method disclosed in the Patent Reference 2, when a resized image is obtained from a source image containing pixels of different colors where an array of RGB colors has a periodicity, two different resizing processes are performed to luminance data and color difference data obtained from the source image in place of resizing the RGB color-separated data obtained from the source image.